

<!doctype html>

<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Administration &amp; Capacity Planning &#8212; RTG Tools Operations Manual v3.12</title>
    <link rel="stylesheet" href="_static/bizstyle.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    
    <script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
    <script src="_static/jquery.js"></script>
    <script src="_static/underscore.js"></script>
    <script src="_static/doctools.js"></script>
    <script src="_static/language_data.js"></script>
    <script src="_static/bizstyle.js"></script>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Appendix" href="appendix.html" />
    <link rel="prev" title="RTG product usage - baseline progressions" href="product_usage.html" />
    <meta name="viewport" content="width=device-width,initial-scale=1.0">
    <!--[if lt IE 9]>
    <script src="_static/css3-mediaqueries.js"></script>
    <![endif]-->
  </head><body>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="appendix.html" title="Appendix"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="product_usage.html" title="RTG product usage - baseline progressions"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">RTG Tools Operations Manual v3.12</a> &#187;</li>
        <li class="nav-item nav-item-this"><a href="">Administration &amp; Capacity Planning</a></li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="administration-capacity-planning">
<h1>Administration &amp; Capacity Planning<a class="headerlink" href="#administration-capacity-planning" title="Permalink to this headline">¶</a></h1>
<div class="section" id="advanced-installation-configuration">
<h2>Advanced installation configuration<a class="headerlink" href="#advanced-installation-configuration" title="Permalink to this headline">¶</a></h2>
<p>RTG software can be shared by a group of users by installing on a
centrally available file directory or shared drive. Assignment of
execution privileges can be determined by the administrator, independent
of the software license file. For commercial users, the software license
prepared by Real Time Genomics (<code class="docutils literal notranslate"><span class="pre">rtg-license.txt</span></code>) need only be included
in the same directory as the executable (<code class="docutils literal notranslate"><span class="pre">RTG.jar</span></code>) and the run-time
scripts (<code class="docutils literal notranslate"><span class="pre">rtg</span></code> or <code class="docutils literal notranslate"><span class="pre">rtg.bat</span></code>).</p>
<p>During installation on Unix systems, a configuration file named
<code class="docutils literal notranslate"><span class="pre">rtg.cfg</span></code> is created in the installation directory. By editing this
configuration file, one may alter further configuration variables
appropriate to the specific deployment requirements of the organization.
On Windows systems, these variables are set in the <code class="docutils literal notranslate"><span class="pre">rtg.bat</span></code> file in the
installation directory. These configuration variables include:</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 11%" />
<col style="width: 89%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Variable</p></th>
<th class="head"><p>Description</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_MEM</span></code></p></td>
<td><p>Specify the maximum memory for Java run-time execution. Use a G suffix for gigabytes, e.g.: <code class="docutils literal notranslate"><span class="pre">RTG_MEM=48G</span></code>. The default memory allocation is 90% of system memory.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_JAVA</span></code></p></td>
<td><p>Specify the path to Java (default assumes current path).</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_JAR</span></code></p></td>
<td><p>Indicate the path to the <code class="docutils literal notranslate"><span class="pre">RTG.jar</span></code> executable (default assumes current path).</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_JAVA_OPTS</span></code></p></td>
<td><p>Provide any additional Java JVM options.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_DEFAULT_THREADS</span></code></p></td>
<td><p>By default any RTG module with a <code class="docutils literal notranslate"><span class="pre">--threads</span></code> parameter will automatically use the number of cores as the number of threads. This setting makes the specified number the default for the <code class="docutils literal notranslate"><span class="pre">--threads</span></code> parameter instead.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_PROXY</span></code></p></td>
<td><p>Specify the http proxy server for TalkBack exception management (default is no http proxy).</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_TALKBACK</span></code></p></td>
<td><p>Send log files for crash-severity exception conditions (default is true, set to false to disable).</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_USAGE</span></code></p></td>
<td><p>If set to true, enable simple usage logging.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_USAGE_DIR</span></code></p></td>
<td><p>Destination directory when performing single-user file-based usage logging.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_USAGE_HOST</span></code></p></td>
<td><p>Server URL when performing server-based logging.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_USAGE_OPTIONAL</span></code></p></td>
<td><p>May contain a comma-separated list of the names of optional fields to include in usage logging (when enabled). Any of <code class="docutils literal notranslate"><span class="pre">username</span></code>, <code class="docutils literal notranslate"><span class="pre">hostname</span></code> and <code class="docutils literal notranslate"><span class="pre">commandline</span></code> may be set here.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_REFERENCES_DIR</span></code></p></td>
<td><p>Specifies an alternate directory containing metagenomic pipeline reference datasets.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">RTG_MODELS_DIR</span></code></p></td>
<td><p>Specifies an alternate directory containing AVR models.</p></td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="run-time-performance-optimization">
<h2>Run-time performance optimization<a class="headerlink" href="#run-time-performance-optimization" title="Permalink to this headline">¶</a></h2>
<p><strong>CPU</strong> — Multi-core operation finishes jobs faster by processing
multiple application threads in parallel. By default RTG uses all
available cores of a multi-processor server node. With a command line
parameter setting, RTG operation can be limited to a specified number of
cores if desired.</p>
<p><strong>Memory</strong> — Adding more memory can improve performance where very high
read coverage is desired. RTG creates and uses indexes to speed up
genomic data processing. The more RAM you have, the more reads you can
process in memory in a run. We use 48 GB as a rule of thumb for
processing human data. However, a smaller number of reads can be
processed in as little as 2 GB.</p>
<p><strong>Disk Capacity</strong> — Disk requirements are highly dependent on the size of the
underlying data sets, the amount of information needed to hold quality
scores, and the number of runs needed to investigate the impact of
varying levels of sensitivity. Though all data is handled and stored in
compressed form by default, a realistic minimum disk size for handling human
data is 1 TB. As a rule of thumb, for every 2 GB of input read data
expect to add 1 GB of index data and 1 GB of output files per run.
Additionally, leave another 2 GB free for temporary storage during
processing.</p>
</div>
<div class="section" id="alternate-configurations">
<h2>Alternate configurations<a class="headerlink" href="#alternate-configurations" title="Permalink to this headline">¶</a></h2>
<p><strong>Demonstration system</strong> — For training, testing, demonstrating,
processing and otherwise working with smaller genomes, RTG works just
fine on a newer laptop system with an Intel processor. For example,
product testing in support of this documentation was executed on a
MacBook PC (Intel Core 2 Duo processor, 2.1 GHz clock speed, 1
processor, 2 cores, 3 MB L2 Cache, 4 GB RAM, 290 GB 5400 RPM Serial-ATA
disk)</p>
<p><strong>Clustered system</strong> — The comparison of genomic variation on a large
scale demands extensive processing capability. Assuming standard CPU
hardware as described above, scale up to meet your institutional or
major product needs by adding more rack-mounted boards and blades into
rack servers in your data center. To estimate the number of cores
required, first estimate the number of jobs to be run, noting size and
sensitivity requirements. Then apply the appropriate benchmark figures
for different size jobs run with varying sensitivity, dividing the
number of reads to be processed by the reads/second/core.</p>
</div>
<div class="section" id="exception-management-talkback-and-log-file">
<h2>Exception management - TalkBack and log file<a class="headerlink" href="#exception-management-talkback-and-log-file" title="Permalink to this headline">¶</a></h2>
<p>Many RTG commands generate a log file with each run that is saved to the
results output directory. The contents of the file contain lists of job
parameters, system configuration, and run-time information.</p>
<p>In the case of internal exceptions, additional information is recorded
in the log file specific to the problem encountered. Fatal exceptions
are trapped and notification is sent to Real Time Genomics with a copy
of the log file. This mechanism is called TalkBack and uses an embedded
URL to which RTG sends the report.</p>
<p>The following sample log displays the software version information,
parameter list, and run-time progress.</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>2009-09-05 21:38:10 RTG version = v2.0b build 20013 (2009-10-03)
2009-09-05 21:38:10 java.runtime.name = Java(TM) SE Runtime Environment
2009-09-05 21:38:10 java.runtime.version = 1.6.0_07-b06-153
2009-09-05 21:38:10 os.arch = x86_64
2009-09-05 21:38:10 os.freememory = 1792544768
2009-09-05 21:38:10 os.name = Mac OS X
2009-09-05 21:38:10 os.totalmemory = 4294967296
2009-09-05 21:38:10 os.version = 10.5.8
2009-09-05 21:38:10 Command line arguments: [-a, 1, -b, 0, -w, 16, -f, topn, -n, 5, -P, -o, pflow, -i, pfreads, -t, pftemplate]
2009-09-05 21:38:10 NgsParams threshold=20 threads=2
2009-09-05 21:39:59  Index[0] memory performance
</pre></div>
</div>
<p>TalkBack may be disabled by adding <code class="docutils literal notranslate"><span class="pre">RTG_TALK_BACK=false</span></code> to the
<code class="docutils literal notranslate"><span class="pre">rtg.cfg</span></code> configuration file (Unix) or the <code class="docutils literal notranslate"><span class="pre">rtg.bat</span></code> file (Window) as
described in <a class="reference internal" href="#advanced-installation-configuration"><span class="std std-ref">Advanced installation configuration</span></a>.</p>
</div>
<div class="section" id="usage-logging">
<h2>Usage logging<a class="headerlink" href="#usage-logging" title="Permalink to this headline">¶</a></h2>
<p>RTG has the ability to record simple command usage information for
submission to Real Time Genomics. The first time RTG is run (typically
during installation), the user will be asked whether to enable usage
logging. This information may be required for customers with a
pay-per-use license. Other customers may choose to send this information
to give Real Time Genomics feedback on which commands and features are
commonly used or to locally log RTG command use for their own analysis.</p>
<p>A usage record contains the following fields:</p>
<ul class="simple">
<li><p>Time and date</p></li>
<li><p>License serial number</p></li>
<li><p>Unique ID for the run</p></li>
<li><p>Version of RTG software</p></li>
<li><p>RTG command name, without parameters (e.g. map)</p></li>
<li><p>Status (Started / Failed / Succeeded)</p></li>
<li><p>A command-specific field (e.g. number of reads)</p></li>
</ul>
<p>For example:</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>2013-02-11 11:38:38007   4f6c2eca-0bfc-4267-be70-b7baa85ebf66    RTG Core v2.7 build d74f45d (2013-02-04)   format  Start   N/A
</pre></div>
</div>
<p>No confidential information is included in these records. It is possible
to add extra fields, such as the user name running the command, host
name of the machine running the command, and full command-line
parameters, however as these fields may contain confidential
information, they must be explicitly enabled as described in <a class="reference internal" href="#advanced-installation-configuration"><span class="std std-ref">Advanced installation configuration</span></a>.</p>
<p>When RTG is first installed, you will be asked whether to enable user
logging. Usage logging can also be manually enabled by editing the
<code class="docutils literal notranslate"><span class="pre">rtg.cfg</span></code> file (or <code class="docutils literal notranslate"><span class="pre">rtg.bat</span></code> file on Windows) and setting
<code class="docutils literal notranslate"><span class="pre">RTG_USAGE=true</span></code>. If the <code class="docutils literal notranslate"><span class="pre">RTG_USAGE_DIR</span></code> and <code class="docutils literal notranslate"><span class="pre">RTG_USAGE_HOST</span></code>
settings are empty, the default behavior is to directly submit usage
records to an RTG hosted server via HTTPS. This feature requires the
machine running RTG to have access to the Internet.</p>
<p>For cases where the machines running RTG do not have access to the
Internet, there are two alternatives for collecting usage information.</p>
<div class="section" id="single-user-single-machine">
<h3>Single-user, single machine<a class="headerlink" href="#single-user-single-machine" title="Permalink to this headline">¶</a></h3>
<p>Usage information can be recorded directly to a text file. To enable
this option, edit the <code class="docutils literal notranslate"><span class="pre">rtg.cfg</span></code> file (or <code class="docutils literal notranslate"><span class="pre">rtg.bat</span></code> file on Windows),
and set the <code class="docutils literal notranslate"><span class="pre">RTG_USAGE_DIR</span></code> to the name of a directory where the user
has write permissions. For example:</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>RTG_USAGE=true
RTG_USAGE_DIR=/opt/rtg-usage
</pre></div>
</div>
<p>Within this directory, the RTG usage information will be written to a
text file named after the date of the current month, in the form
<code class="docutils literal notranslate"><span class="pre">YYYY-MM.txt</span></code>. A new file will be created each month. This text file can
be manually sent to Real Time Genomics when requested.</p>
</div>
<div class="section" id="multi-user-or-multiple-machines">
<h3>Multi-user or multiple machines<a class="headerlink" href="#multi-user-or-multiple-machines" title="Permalink to this headline">¶</a></h3>
<p>In this case, a local server can be started to collect usage information
from compute nodes and recorded to local files for later manual
submission. To configure this method of collecting usage information,
edit the <code class="docutils literal notranslate"><span class="pre">rtg.cfg</span></code> file (or <code class="docutils literal notranslate"><span class="pre">rtg.bat</span></code> file on Windows), and set the
<code class="docutils literal notranslate"><span class="pre">RTG_USAGE_DIR</span></code> to the name of a directory where the local server will
store usage logs, and <code class="docutils literal notranslate"><span class="pre">RTG_USAGE_HOST</span></code> to a URL consisting of the name
of the local machine that will run the server and the network port on
which the server will listen. For example if the server will be run on a
machine named <code class="docutils literal notranslate"><span class="pre">gridhost.mylan.net</span></code>, listening on port <code class="docutils literal notranslate"><span class="pre">9090</span></code>, writing
usage information into the directory <code class="docutils literal notranslate"><span class="pre">/opt/rtg-usage/</span></code>, set:</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>RTG_USAGE=true
RTG_USAGE_DIR=/opt/rtg-usage
RTG_USAGE_HOST=http://gridhost.mylan.net:9090/
</pre></div>
</div>
<p>On the machine <code class="docutils literal notranslate"><span class="pre">gridhost</span></code>, run the command:</p>
<p>$ rtg usageserver</p>
<p>Which will start the local usage server listening. Now when RTG commands
are run on other nodes or as other users, they will submit usage records
to this sever for collation.</p>
<p>Within the usage directory, the RTG usage information will be written to
a text file named after the date of the current month, in the form
<code class="docutils literal notranslate"><span class="pre">YYYY-MM.txt</span></code>. A new file will be created each month. This text file can
be manually sent to Real Time Genomics when requested.</p>
</div>
<div class="section" id="advanced-usage-configuration">
<h3>Advanced usage configuration<a class="headerlink" href="#advanced-usage-configuration" title="Permalink to this headline">¶</a></h3>
<p>If you wish to augment usage information with any of the optional
fields, edit the <code class="docutils literal notranslate"><span class="pre">rtg.cfg</span></code> file (or <code class="docutils literal notranslate"><span class="pre">rtg.bat</span></code> file on Windows) and set
the <code class="docutils literal notranslate"><span class="pre">RTG_USAGE_OPTIONAL</span></code> to a comma separated list containing any of
the following:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">username</span></code> - adds the username of the user running the RTG command.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">hostname</span></code> - adds the machine name running the RTG command.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">commandline</span></code> - adds the command line, including parameters, of the
RTG command (this field will be truncated if the length exceeds 1000
characters).</p></li>
</ul>
<p>For example:</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>RTG_USAGE_OPTIONAL=username,hostname,commandline
</pre></div>
</div>
</div>
</div>
<div class="section" id="command-line-color-highlighting">
<h2>Command-line color highlighting<a class="headerlink" href="#command-line-color-highlighting" title="Permalink to this headline">¶</a></h2>
<p>Some RTG commands make use of ANSI colors to visually enhance terminal
output, and the decision as to whether to colorize the output is
automatically determined, although some commands also contain additional
flags to control colorization.</p>
<p>The default behavior of output colorization can be configured by
defining a Java system property named <code class="docutils literal notranslate"><span class="pre">rtg.default-markup</span></code> with an
appropriate value and supplying it via RTG_JAVA_OPTS. For example, to
disable output colorization, use:</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>RTG_JAVA_OPTS=&quot;-Drtg.default-markup=none&quot;
</pre></div>
</div>
<p>The possible values for <code class="docutils literal notranslate"><span class="pre">rtg.default-markup</span></code> are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">auto</span></code> - automatically enable ANSI markup when running on
non-Windows OS and when I/O is detected to be a console.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">none</span></code> - disable ANSI markup.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ansi</span></code> - enable ANSI markup. This may be useful if you are using
Windows OS and have installed an ANSI-capable terminal such as
ANSICON, ConEmu or Console 2.</p></li>
</ul>
</div>
</div>


            <div class="clearer"></div>
          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
            <p class="logo"><a href="index.html">
              <img class="logo" src="_static/html_logo.png" alt="Logo"/>
            </a></p>
  <h3><a href="index.html">Table of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">Administration &amp; Capacity Planning</a><ul>
<li><a class="reference internal" href="#advanced-installation-configuration">Advanced installation configuration</a></li>
<li><a class="reference internal" href="#run-time-performance-optimization">Run-time performance optimization</a></li>
<li><a class="reference internal" href="#alternate-configurations">Alternate configurations</a></li>
<li><a class="reference internal" href="#exception-management-talkback-and-log-file">Exception management - TalkBack and log file</a></li>
<li><a class="reference internal" href="#usage-logging">Usage logging</a><ul>
<li><a class="reference internal" href="#single-user-single-machine">Single-user, single machine</a></li>
<li><a class="reference internal" href="#multi-user-or-multiple-machines">Multi-user or multiple machines</a></li>
<li><a class="reference internal" href="#advanced-usage-configuration">Advanced usage configuration</a></li>
</ul>
</li>
<li><a class="reference internal" href="#command-line-color-highlighting">Command-line color highlighting</a></li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="product_usage.html"
                        title="previous chapter">RTG product usage - baseline progressions</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="appendix.html"
                        title="next chapter">Appendix</a></p>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="_sources/administration.rst.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="appendix.html" title="Appendix"
             >next</a> |</li>
        <li class="right" >
          <a href="product_usage.html" title="RTG product usage - baseline progressions"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">RTG Tools Operations Manual v3.12</a> &#187;</li>
        <li class="nav-item nav-item-this"><a href="">Administration &amp; Capacity Planning</a></li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &#169; Copyright 2017, Real Time Genomics.
      Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 3.1.2.
    </div>
  </body>
</html>